In the contexts of computer-aided design (CAD), computer-aided engineering (CAE), computer-Aided Manufacturing (CAM), virtual worlds, online video games, and more generally in the context of 3D online applications, the display of a three-dimensional (3D) environment (also referred to as 3D scene) and its content—various assets that describe a partial or complete 3D virtual environment—is still a challenge. The 3D scene comprise 3D models, wherein each 3D model is an asset and has a usual representation including polygonal meshes, spline surfaces, constructive solid geometries . . . . Materials and textures can be also assets of the 3D scene, as well as animations of the 3D models, lights, scene graphs. The computer hosting the assets is referred to as a content server. A rendered image of this 3D scene viewed from a particular viewpoint (also referred to as virtual camera) can be displayed on a second computer. This process of remote rendering is typically used for cloud gaming, virtual globes, virtual world. The second computer on which this remote rendering is carried out is referred to as a client.
A 3D online application includes a software component called 3D rendering engine to produce one or more pictures of the 3D scene viewed from a particular viewpoint by taking into account 3D projections and several lighting effects (shadows, reflections, etc.). To achieve its goal, such 3D online application must transfer information of the 3D scene from the server(s) to the client(s) at given times.
There are basically two scenarios for performing the remote rendering of the 3D scene. The first one consists in directly sending 3D content from a server to a client and this client performs the rendering step. This will be referred as geometry streaming. This method usually describes how virtual worlds work. The second scenario comprises a server that performs the rendering step and sends the result (static image or video streams) to a client. This will be referred as pixel streaming. This method usually describes how remote rendering and cloud gaming work.
Geometry streaming can use many strategies to transmit the content of a 3D scene. An example of strategy is the concept of level of detail: for each asset a minimum quality level, that does not disturb the overall quality of the final rendering, is deduced from the visibility computation. If the version of an asset in the client cache does not reach the required quality level, the server sends refinements (usually the differences between the required version and the version in the client cache. Examples of involved technologies in such strategy are progressive meshes, terrain displacement mapping, voxel based point cloud, hierarchical levels of detail.
In the case of pixel streaming transferred data are pictures or videos usually compressed with standard compression schemes (JPEG/JPEG2000 for images and H.264 for videos). Some variants exist to distribute the computation across multiple computers. In 3D computer graphics it refers to the 2D rectangle used to project the 3D scene to the position of a virtual camera. A viewport is a region of the screen used to display a portion of the total image to be shown. The application can cut the viewport and assign the responsibility of rendering each section (smaller frames) to several computers. In another variant, the application can cut the 3D scene and assign the responsibility of rendering each section (smaller sets of assets) to several computers.
Geometry streaming and pixel streaming handle differently the issues encountered with 3D online applications. They have their own disadvantages and their own limit as explained now.
Geometry streaming and pixel streaming differently scale with an increasing number of clients. Pixel streaming is heavily penalized, since a big part of the computation (including the rendering) is made on the server side. Thus, the number of required servers quickly grows with the number of clients. It is hard to find computation results that can be reused from one client to another since they rarely share the same point of view. Geometry streaming scales better since it requires less computation on the server side. However, sending over a computer network geometries consumes bandwidth of the network and generally involves lags—less reactivity for the application—on the 3D scene rendered on the client computer.
Geometry streaming and pixel streaming involve client hardware requirements. As mentioned above, pixel streaming has only a small computation cost on the client side, while geometry streaming requires some 3D display capabilities on the client side. Although most of light devices now benefit from hardware acceleration for 3D display, their capabilities may be relatively limited, which may lead to less realistic pictures or less reactivity for the application.
Both geometry streaming and pixel streaming have a significant bandwidth consumption and require network reliability. Many efficient rate-distortion algorithms can be used with pixel streaming to ensure a fair consumption of the bandwidth. However streaming videos is usually quite expensive compared to other applications. If the available bandwidth is suddenly lowered, the quality of the transmitted images can be reduced accordingly. In case of network interruption, the client will not be able to change its viewpoint or to interact with the 3D scene. As introduced previously, geometry streaming is more likely to experience spikes in the bandwidth usage, leading to bandwidth saturation and network errors. An additional effort must be done on the design of the application (use of levels of detail or LODs) and on the design of the 3D scene (the amount of information in different regions of the scene graph must be balanced) to mitigate this effect. If the available bandwidth is suddenly lowered, bandwidth saturation will occur more, often leading to missing/deteriorated parts of the scene in the client view. In case of network interruption, the client is still able to change its viewpoint and to interact with the elements of the scene that were already transmitted.
Pixel streaming does not allow performing operations on/between the 3D modeled objects displayed on the client such as collision detection, display of visual hints for clashes, snapping objects to each other or simulating rigid body contacts. Indeed, the client does not have the 3D content of the 3D modeled object and only renders a representation computed by the server. Geometry streaming allows the client to perform such operation; however, the computing resources of the client are limited and they are already used for rendering the 3D scene. Contact and collision detection algorithms are well developed: for example in video games, the need to simulate solid dynamics is now common and is done by a component called physics engine. This component needs to perform several tasks including the collision detection.
Within this context, there is still a need for an improved method for performing a rendering of a 3D scene on a client connected to a server. Preferably, the remote rendering allows to perform operations on/between the 3D modeled objects of the 3D scene.